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EXECUTIVE SUMMARY 


Higher education is facing a host of challenges, including external questions regarding its value 
and purpose. These questions cut to the core of the states’ role in higher education. Traditionally, 
states have the responsibility to ensure that institutions of higher education are operating in the 
public interest and that the institutions are good stewards of their public resources. Central to 
this responsibility is the question of institutional and educational quality. Concerns regarding 
higher education quality and the states’ role in quality assurance and improvement motivated 
the State Higher Education Executive Officers Association (SHEEO) and the National Association 
of System Heads (NASH) to partner with Lumina Foundation to investigate current state and 
system practices and to work toward recommendations for future action. 


Using a variety of data sources, including two in-person convenings of relevant stakeholders, 
a survey of state higher education agencies and system offices, and qualitative interviews, 
the partners explored current quality assurance and improvement practices, challenges and 
limitations related to quality assurance and improvement at the state and system levels, and 
ideas for how current policies and practices might be improved. 


The findings revealed that approaches are varied and limited by a lack of consensus around 
what quality means in higher education and what the appropriate level and manner of state 
and system engagement regarding quality should be, and limited resources (staff, money, 
and technology). Nevertheless, participants and respondents provided important insights into 
how states and systems might better engage in the question of quality and work to assure and 
improve quality in higher education. In that regard, we recommend the following: 


Arrive at widely agreed upon understandings of quality. While state agencies and system offices 
will and ought to have their specific definitions of quality, developing shared understandings 
of quality would help individuals and entities advance our shared work in defining, assessing, 
and improving quality. A collaborative effort across states and with other stakeholders with 
interest in and responsibility for aspects of quality assurance will be needed to arrive at a shared 
understanding. We believe the benefits warrant the effort. 


Develop a greater understanding among all relevant actors of the state's interest and role 
in educational quality. There is still some lack of knowledge and appreciation of the state's 
specific role in quality assurance and improvement. This lack of understanding exists among 
institutional and faculty leaders, accreditors, the federal government, and state and system 
policy leaders and practitioners. Additional thinking and work are needed to properly articulate 
and defend this state role and to explain why and how states ought to be involved in this work. 


Identify best practices in quality assurance. Additional efforts are needed to identify what works 
in quality assurance and improvement and to diffuse those efforts across states and systems. 
This will require engagement from state and system leaders, college and university leaders, 
faculty, academic researchers, think tanks, policy organizations, and funders. 


Make program review and state authorization meaningful quality assurance processes. 
Essential roles of state agencies and/or system offices—program review and state authorization— 
are often treated as bureaucratic processes and even rubber stamps. Making these processes 
substantive and focused on quality assurance is challenging but necessary. Ideas for improvement 
include requiring institutions to submit assessable learning outcomes, descriptions of how they 
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will meet those outcomes in meaningful ways, plans for faculty development, engaging in 
follow-up reviews of authorized institutions and approved programs, and requiring evidence of 
the institutions’ or programs’ success in accomplishing the approved student learning outcomes. 
Continuing to assess the institution's capacity and resources and other existing requirements will 
likewise be valuable. Further, state and system leaders, researchers, faculty, and other interested 
parties ought to consider what innovations in state authorization and program approval might 
help improve quality assessment and improvement efforts. 


Treat equity as a quality consideration. Among participants and respondents, there appeared 
to be a coalescing around the need to close equity gaps and to treat equity considerations as 
quality considerations. However, better data, research, and political and institutional will are 
needed to properly address quality concerns from an equity perspective. Understanding gaps in 
student resources and opportunities and measuring the ability of institutions to improve higher 
education access and outcomes for underserved students will be critical in considering how 
to address equity gaps. Such examinations need to be done within and across institutions. A 
quality system of higher education needs to be quality for all students, not just well resourced, 
white, and high achieving students. In that regard, states have a particular responsibility for 
looking out for underserved students and ensuring they receive a quality education. Quality 
public institutions should, in the state's eyes, deliberately promote economic mobility and 
opportunity and work to close achievement gaps. Again, collaborative efforts between state 
and system leaders, college and university leaders, academic researchers, faculty, think tanks, 
policy organizations, and funders will be needed. 


Actively engage faculty and institutional leaders. Ultimately, a quality education is dependent 
on what happens within our colleges and universities and in particular what happens in our 
classrooms. Actively seeking the participation of institutional leaders and faculty will be critical 
to the success of any quality assurance and improvement efforts. State agencies and systems 
ought to develop mechanisms for the inclusion of faculty and institutional leaders in the 
state- and system-level quality assurance and improvement efforts such as working groups 
and advisory boards. They ought to also develop relationships with individual faculty members 
and institutional leaders that allow for state and system leaders to seek ad hoc information, 
reactions, and advice. State agencies and system offices also ought to explore ways they might 
support faculty in their critical role. 


Invest in data, tools, and people. State lawmakers will need to provide state higher education 
agencies and system offices with the financial resources to collect the appropriate data, 
implement the appropriate assessments, conduct the appropriate analyses, properly interpret 
the information, and design and implement the appropriate policies and practices. This will 
require hiring qualified staff and investing in the necessary infrastructure and tools. 


Open lines of communication and real partnership between members of the triad. As described 
and authorized in the federal Higher Education Act, the U.S. Department of Education, approved 
accreditors, and state governments make up the program integrity triad. These three entities 
are supposed to work together to ensure the quality of providers of postsecondary credentials. 
However, the triad has not always functioned appropriately or lived up to its obligations. 
Opening lines of communication, developing agreed upon protocols for information and data 
collection and sharing, developing shared understandings and agreements regarding roles and 
responsibilities, and engaging in more collaborative work and peer learning would all help the 
triad function more effectively. 
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QUALITY ASSURANCE AND IMPROVEMENT 
IN HIGHER EDUCATION 


THE ROLE OF THE STATES 


Higher education is facing a host of internal and external challenges, including constrained 
resources resulting from reductions in state support, competition from nontraditional providers 
of training and credentials, and in many states, a declining number of high school graduates. 
Further, since the beginning of 2016, at least 104 colleges and universities have closed, not 
counting acquisitions, consolidations, and mergers. Several of these have been large national 
for-profit chains affecting thousands of students in states across the country. 


As a result of these challenges, there is a growing suspicion of and lack of confidence in higher 
education generally. Questions regarding the quality of the educational experience provided 
to students and the value of their credentials abound. Less than half (48 percent) of Americans 
express “a great deal” or “quite a lot" of confidence in higher education. This represents a 9 
percentage point decline from 2015, and among Republicans, confidence has declined by 17 
percentage points. Central to these challenges and a primary concern is the question of quality 
and what students get for the money and time they commit to higher education. Likewise, 
growing evidence indicates that resources, opportunities, and educational experiences are not 
equally distributed, with low-income, students of color, and rural students segregated into lower 
resourced institutions, for-profit institutions, and institutions with lower educational outcomes.* 


States play a fundamental role in higher education, protecting both the investment of state 
dollars and their citizens as their consumers. Despite their central responsibility in this regard, 
states have struggled with how they might help ensure that students are receiving a quality 
educational experience. Definitions of quality in higher education are varied and contested, while 
measuring quality may be an even more difficult task. Even where definitions and measures exist, 
it can be challenging to know how states might use them in actual quality improvement efforts. 
To help address these challenges, the State Higher Education Executive Officers Association 
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(SHEEO), the National Association of System Heads (NASH), and Lumina Foundation partnered 
to conduct an environmental scan of the existing landscape of approaches utilized to assess 
and assure the quality of higher education institutions and credentials at the state and system 
levels. This white paper reports on the findings of this project. 


PROJECT AND REPORT OVERVIEW 


With funding from Lumina Foundation, SHEEO and NASH conducted an environmental scan of 
the existing landscape of approaches utilized to assess and assure the quality of higher education 
institutions and higher education credentials at the state and system levels. The project was 
divided into four phases: 


Phase 1: The partners convened with a group that included but extended beyond the 
traditional program integrity triad, made up of states, the federal government, and 
accreditors. This convening included state higher education executive officers (SHEEOs), 
university system leaders, regional and national accreditors, a representative from the U.S. 
Department of Education, and other interested parties to discuss quality assurance and 
improvement and ideas for strengthening the triad. 


Phase 2: The partners contracted with the Indiana University Center for Postsecondary 
Research (IUCPR) to help design, administer, and analyze the data from a survey designed 
to learn about SHEEO and NASH members’ definitions, activities, and experiences with 
regard to quality assurance and improvement in their states, including current practices. 


Phase 3: The partners conducted interviews with state higher education agency and 
university system leaders to probe more in-depth into their quality assurance and 
improvement efforts, including definitions, capacities, and engagements. The partners 
then contracted with Research for Action (RFA) to code and analyze the qualitative data. 


Phase 4: The partners reconvened with representatives from state higher education agencies 
and system offices, institutional leaders, academic researchers, policy organization staff, 
accreditors, think tanks, and others to review the findings from the first three phases of the 
project and share promising quality assurance practices. This group also considered changes 
to current practices and divisions of responsibility regarding quality assurance. 


Using the data from the four research phases of this project and the reports authored by IUCPR 
and RFA, this report summarizes the findings of the project. The findings are often disaggregated 
using the broad categories of state agency and system office. For this report, the state category 
refers to those respondents from state-level bodies (e.g., coordinating and planning boards) 
that did not classify themselves as belonging to a system office. System offices often have direct 
operating, administrative, and financial control over the institutions they oversee. Likewise, they 
generally oversee a specific sector of institutions (public universities or community colleges). 
State agencies generally have less operational control and instead must often rely on state 
policy changes and/or persuasion to see changes in institutional actions. 


The findings are divided into six critical topics that rose to prominence across our data sources. 
These include (1) definitions and assessment of quality, (2) quality improvement efforts, (3) equity 
and quality, (4) the program integrity triad, (5) challenges and limitations in states’ quality efforts, 
and (6) future directions and recommendations for state higher education agencies and system 
offices. We hope the findings from this project will inform the field and generate momentum to 
improve quality assurance efforts at the state and system levels. 
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DEFINING AND ASSESSING QUALITY 


2: 


As seen in Table 1, the majority of state agency’ and system office respondents place quality 
assurance as an essential priority within the mission of the organization. However, both the survey 
data and the interview data revealed modest consensus, at best, in how these entities define 
quality in higher education. Several survey and interview respondents reported that their offices 
had no operating definition of quality. As articulated in RFA’s report from the interview data, the 
definitions that were provided can be placed into two broad categories: (1) the abstract and 
conceptual and (2) the operationalized. The abstract and conceptual definitions generally related 
to ideas of personal growth and students accomplishing their full potential. The operationalized 
definitions were more tied to specific measures, such as employment outcomes, graduation 
rates, equitable graduation rates across student groups, licensure, and certification exam scores. 
In both sets of data, several agencies and offices deferred to accreditation both as a definition 
and a measure of quality. Others had no conceptual or operationalized definition of quality. 


TABLE 1: 
WHERE QUALITY ASSURANCE FITS IN THE OVERALL MISSION OF THE ORGANIZATION 


A top priority (central to our mission) 14.30% 12.50% 13.60% 
A priority, but co-equal with other important functions and concerns 57.10% 50.00% 54.50% 
Important, but other concerns have higher priority 28.60% 37.50% 31.80% 
Not important or outside our purview 0% 0% 0% 


Notes: Respondents were instructed: “Please indicate where quality assurance fits in the overall mission of your organization.” 


In the survey, respondents were asked to identify from a list of education outputs, priorities, and 
impacts the factors they considered to be important in defining quality at the state or system level 
(see Figure 1). They were then asked to identify the top three among their selections (see Figure 
2). Across both questions, while occurring in different orders depending on the question, top 
factors were: reducing attainment gaps, undergraduate degree production, talent development, 
and student learning outcomes. Differences are apparent between states and systems in their 
ranking of the importance of talent development and retention in-state, with states giving far 
greater importance to that factor than systems (Figures 1 and 2). Likewise, as seen in Figure 2, 
the state ranked student learning outcomes higher than did systems. 


In the survey results, “state” refers to those respondents from state-level governing, coordinating, and planning bodies that did not 
classify themselves as belonging to a system office. 
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FIGURE 1: 


FACTORS IN DETERMINING HIGHER EDUCATION QUALITY DEFINITIONS: 
PROBABILITY OF MENTION BY TYPE OF ORGANIZATION 
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Probing more deeply into these questions and concepts in the interviews revealed a distinction 
between what might be called academic quality and institutional quality. Academic quality was 
generally understood as academic and learning outcomes that respondents defined as the 
core of quality higher education. While inclusive of academic quality, institutional quality was 
understood as broader and included sector-specific and institution-level factors and concepts. 
In their report, based on the qualitative data, RFA broke down the concepts and their associated 
outcomes in this way: 
e Academic quality: 
- Quality instruction; 
— Student learning; 
— Employability or workforce currency; 
— Licensure attainment; 
— Alumni satisfaction; 
- Accomplishment of student goals (such as transfer or personal growth); and 
- Faculty quality, faculty/student ratio, and faculty diversity. 
e Institutional quality: 
- Institutional performance, including: 
- Rates of student enrollment, retention, and outcomes; 
- Presidential/leadership quality; 
- Technology and facilities; 
- The financial health or sustainability of the institution; and 
- Affordability or manageable debt-to-income ratio for students. 
- Sector-specific indicators of quality, including: 
- Research production for research universities; and 


- Alignment to local workforce demands, comprehensive universities, and 
community colleges. 


-— Quality contributions to the state or community, including: 


- Equitable student access and outcomes for underserved student 
populations; 


- Economic mobility for students; 
- Civic engagement of alumni; 
- Economic contributions to the state or community; and 


- Commitment to community well-being. 
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MEASURES OF QUALITY 


Regarding specific quality measures employed by state agencies and system offices, the traditional 
measures of graduation and retention rates were among the most likely to be employed (see 
Figure 3). Transfers were also near the top of the list for system offices as were enrollments by 
sector. Licensure exam performance was frequently mentioned by state agencies. 


FIGURE 3: 
MEASURES USED TO ASSESS THE QUALITY OF HIGHER EDUCATION: 
PROBABILITY OF MENTION BY TYPE OF ORGANIZATION 
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Interviewees also suggested that accreditation served as a measure of quality. Several state 
agency interviewees reported that they defaulted to accreditors when it came to quality 
assessment, feeling that the state ought not to involve itself directly in matters of academic 
quality. For others, accreditation served as a minimum threshold for quality, and they believed 
states and institutions have a responsibility to assure quality above and beyond accreditation. 
Similarly, several interviewees deferred to institutions and specifically to the faculty when it came 
to defining and assuring academic quality. As one chief academic officer explained: “! will let 
faculty do what faculty do. I’m not trying to do their job. My job is to be of service to them.” And 
a SHEEO stated, “That's [faculty members’] job, and we think you're [the faculty] really good at it. 
We also believe that accreditors kind of help to keep [faculty] on the right path, and accreditation 
is a way of checking in.” 


Additional measures for academic quality mentioned in the interviews included: accreditation 
outcomes and measures, program review, alumni satisfaction, and institutional learning 
assessments. For institution quality, interviewees also mentioned: the financial health of the 
institution, externally funded research dollars, and economic and social impact. 


In the interviews, respondents also indicated a need to address institutional quality from a sector- 
specific perspective. In that regard, respondents indicated that quality assessments ought to 
relate to an institution’s mission and sector-specific characteristics. For research universities, 
research production and funding may be important measures. Likewise, respondents expressed 
concern about the quality of for-profit institutions and some of their national accreditors. 


DATA SOURCES FOR MEASURING QUALITY 


Specific data sources used by state agencies and system offices varied. Traditional sources, 
like the Integrated Postsecondary Education Data System (IPEDS) administered by the U.S. 
Department of Education and the National Student Clearinghouse, were the most popular. 
Surprisingly, IPEDS data appeared to be the most popular among the options included in the 
survey, although local in-house data sources were not provided as a choice. IPEDS only includes 
aggregate institution-level data, which tend to be over a year old and are publicly available. 
Likewise, IPEDS does not include any learning outcomes data. Nevertheless, IPEDS is one of 
the only sources for comparable institution-level data across sectors and states. State agencies 
are far more likely to use state student-level data and are likewise more likely to make use 
of state unemployment data, which provide data on employment outcomes (e.g., salary and 
employment area). 
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FIGURE 4: 
DATA SOURCES USED TO ASSESS QUALITY OF UNDERGRADUATE EDUCATION: 
PROBABILITY OF MENTION BY TYPE OF ORGANIZATION 
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Other data sources mentioned in the interviews included: licensure exams, loan default rates, 
institution economic impact reports, Bureau of Labor Statistics, accreditation and program 
review reports, Gallup Alumni Survey, Equifax graduate outcomes metrics, and institutions’ 
audited financial statements. 


QUALITY IMPROVEMENT EFFORTS 


Among respondents, state agencies appeared to have greater satisfaction with the states’ overall 
quality assurance efforts when compared to system office respondents (see Table 2). The 
majority of state agency (64 percent) and system office (56 percent) respondents were somewhat 
satisfied with the approach they are taking to assure the quality of higher education. However, 
a larger share of state agency respondents were very satisfied (29 percent) when compared 
with system office respondents (11 percent). Likewise, a larger share of system respondents (33 
percent) were somewhat dissatisfied compared to state office respondents (7 percent). Because 
respondents were asked to assess their level of satisfaction with their state’s approach to quality 
assurance, system respondents may have been assessing the state's effort writ large (beyond 
their system office) or those efforts of a specific state higher education agency (again external 
to their system) and not the system's efforts themselves. 
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TABLE 2: 
SATISFACTION WITH THE APPROACHES THE STATES ARE TAKING TO ASSURE 
THE QUALITY OF HIGHER EDUCATION 


Very satisfied 28.60% 11.10% 21.70% 
Somewhat satisfied 64.30% 55.60% 60.90% 
Somewhat dissatisfied 7.10% 33.30% 17.40% 
Very dissatisfied 0% 0% 0% 


Notes: Respondents were asked: “How satisfied are you with the approach your state is taking to assure the quality of higher education?” 


Interview participants described access to sophisticated statewide student-level data that link 
K-12, higher education, and workforce outcomes as critical tools in their quality assurance 
efforts. These participants reported using data to “increase the accessibility and transparency of 
the information that's available to students and employers" and to “catalyze improvements and 
hold institutions accountable for quality outcomes.” 


As seen in Figure 5, the most frequent uses of quality metrics (the metrics asked about in the 
survey) are in communication with institutional leadership, in accountability systems, and with 
accreditation-related work. Use in program approval is also common in both state agency and 
system offices. State agency respondents were more likely to report frequently using quality 
metrics in their communication with elected officials. 
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FIGURE 5: 
USE OF QUALITY METRICS: PERCENTAGE REPORTING “VERY MUCH" 
OR “QUITE A BIT” BY ORGANIZATION TYPE 
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Note: Respondents were asked: “How much are the metrics you identified used in the following activities of your organization?” 
Options included: very much, quite a bit, some, very little, or not at all. 


Interviewees also reported that they use their quality assurance data and measures to inform their 
strategic planning, to drive their quality improvement funding, and to inform the development 
of equity focused policies. 


The need to collect and address student learning outcomes was a topic of great interest and 
discussion at both convenings. While good examples of utilization of these tools at the state or 
system level were not provided, they are seen as potential resources and areas of development 
for improving quality in academic programs. 
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PROGRAM REVIEW AND APPROVAL 


Program approval was a topic that came up frequently in the qualitative interviews and during 
the final convening. As reported in Figure 5 above, close to 80 percent of survey respondents 
reported using quality assurance metrics in program approval. However, program approval 
is often viewed as a bureaucratic compliance exercise with little connection to other quality 
assurance efforts. Nevertheless, participants felt that it might function as an important quality 
assurance mechanism if approached in the right way. Program approval is often used as a 
way of avoiding unnecessary program duplication and for ensuring that there is demand 
and need for a new program. Participants also argued that the program approval process 
might require institutions to submit assessable learning outcomes and descriptions of how 
the proposed program will meet those outcomes in meaningful ways. Likewise, participants 
argued that during follow-up reviews of approved programs, states and systems might require 
institutions to provide evidence of the programs’ success in accomplishing the approved student 
learning outcomes. 


EQUITY AND QUALITY 


Across the data sources, there appeared to be a growing recognition that equity concerns, 
including achievement gaps, are quality concerns. In fact, in the final convening, there was 
widespread agreement that a program could not be considered a quality program if significant 
achievement gaps existed. As one participant explained: 


Quality assurance and the furtherance of equity are interdependent. Authentic quality 
assurance must promote equity. Authentic efforts to advance equity must incorporate 
attention to quality assurance. And we must aggressively dispel the canard that expansion 
of access is synonymous with lower quality. To the contrary, claims for “academic quality" 
must be regarded as inconsistent with significant equity gaps. 


And another noted their desire to “Work more intentionally and emphatically to help people in 
my state understand that equity and quality are constituent components of one another—these 
two things are not separate endeavors.” 


The vast majority of survey respondents reported disaggregating their quality metrics by 
student populations (e.g., racial/ethnic groups, low-income students, first-generation students, 
nontraditional students) across and within institutions (see Table 2). However, state agencies 
were more likely to only disaggregate some of the metrics. This was particularly true for within 
institution metrics where nearly 31 percent of state agencies reported only disaggregating some 
of the metrics, while all system offices reported either disaggregating all or most of their data by 
student population within institution quality metrics. 
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TABLE 3: 

PERCENTAGE OF QUALITY ASSURANCE METRICS DISAGGREGATED BY STUDENT 
POPULATION (E.G., RACIAL/ETHNIC GROUPS, LOW-INCOME STUDENTS, 
FIRST-GENERATION STUDENTS, NONTRADITIONAL STUDENTS) ACROSS AND 
WITHIN INSTITUTIONS 


53.80% 50.00% 52.40% 
Most 23.10% 37.50% 28.60% 
Some 23.10% 12.50% 19.00% 
None 0% 0% 0% 

53.80% 62.50% 57.10% 
Most 15.40% 37.50% 23.80% 
Some 30.80% 0% 19.00% 
None 0% 0% 0% 


In the interviews, respondents reported a desire to identify and track achievement gaps down 
to the program level. As reported in the RFA report, one system leader explained, “We are just 
now getting into which programs underrepresented students either get into, or never get into, 
and whether or not they're successful.” That system leader went on to explain that even if an 
institutional achievement gap appears to close, “If there's never been an African American male 
ever graduating from a particular program, we still have a problem." 


THE TRIAD 


During the phase-one meeting of members of the program integrity triad (states, accreditors, 
and the federal government), there was widespread agreement that to improve the quality 
assurance function of the triad, members will need to engage in better information sharing 
and communication. Participants pointed to examples of institutions nearing failure or closing 
without all members of the triad knowing of the institutions’ dire circumstances. Participants 
appeared to agree that before one member of the triad takes significant action (accreditation 
visit, placing an institution on accreditation probation, placing an institution on heightened cash 
monitoring, etc.), that member ought to contact the other members of the triad. This would 
provide forewarning and the opportunity to gather additional information and perspective. The 
need for states to do a better job monitoring the financial viability of the institutions within 
their states and to better assess the quality of their institutions and academic programs was 
also discussed. Likewise, there was agreement that greater clarification is needed regarding 
the distribution of responsibilities before and after an institutional closure. Finally, there was 
agreement that an official annual meeting of the members of the triad would be helpful. 
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CHALLENGES AND LIMITATIONS 


Across the data collection efforts, several challenges were repeatedly mentioned. Among those 
included were: 


Emerging credentials. Respondents and participants indicated that new credentials pose a 
significant challenge to quality assurance efforts. State agencies reported that they often do not 
engage in program approval for short-term certificates at their public colleges and universities. 
This may be because they want to give institutions flexibility to be responsive to the marketplace, 
they want to minimize the administrative burden on institutions, or because courses that comprise 
these certificates are also included in approved programs. Also, it was reported that they currently 
make little or no effort to collect information about other credentials like badges, micro- 
credentials, nontraditional certificates, credentials offered by unaccredited and unauthorized 
providers, and the like. Additionally, work-based training and credentialing were also highlighted 
as areas where state higher education agencies and system offices often have little awareness. 
Because of their limited authority, responsibility, understanding, and awareness of these types of 
credentials, participants and respondents agreed that this is an area of significant quality concern. 


Nontraditional providers. Similarly, participants and respondents highlighted nontraditional 
providers of credentials as an area of growing concern. These providers tend to be for-profit 
entities and often do not seek to be eligible for federal student financial aid and, as a result, do 
not seek accreditation. Some even try to avoid going through state authorization. Nevertheless, 
they charge tuition and fees, promising students a return on their investment. State agencies 
were particularly concerned about the quality of these providers. 


Defining quality. Many participants and respondents struggled with the idea of academic or 
institutional quality at the conceptual level. For many, understanding what it is and/or what it 
should be is still a challenge. For others, moving from an idea of quality to a definition of quality 
was Challenging. 


Data, assessments, and measures. The question of how state agencies and system offices 
can accurately assess quality came up repeatedly. Salient issues included: what data ought to 
be collected and how, what assessment tools might provide indications of quality, and what 
measures might be created from the data. The lack of widely accepted standardized assessments 
that can accurately and broadly measure student learning serves as a particular challenge to state 
and system efforts. This was particularly a concern when it came to assessing learning outcomes. 


Balancing state oversight and academic freedom. A number of respondents and participants 
indicated that quality is squarely the responsibility of the faculty. Balancing state oversight and 
academic freedom was a central concern even for those state agencies and system offices who 
reported playing an active role in quality assurance and improvement. Faculty play the central role 
in quality, and so questions regarding how states and systems might support faculty and how they 
might include faculty in the state- and system-level quality assurance and improvement efforts 
are critical. 


Resources and staff. Adequate resources and staff knowledge and skills were frequently cited 
as Challenges in state agencies’ and system offices’ efforts to assess and improve quality in 
higher education. Participants and respondents indicated that their offices lacked the financial 
resources necessary to engage in widespread assessments and/or the technological resources 
to properly use the data and assessments. Further, many indicated that they did not have the staff 
with the appropriate Knowledge or skills necessary to engage in the type of quality assessment 
and improvement activities they would like to engage in. 
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RECOMMENDATIONS AND FUTURE DIRECTIONS 


Based on the data collected through this project, we make the following recommendations: 


Arrive at widely agreed upon understandings of quality. While state agencies and system offices 
will and ought to have their specific definitions of quality, developing shared understandings 
of quality would help individuals and entities advance our shared work in defining, assessing, 
and improving quality. A collaborative effort across states and with other stakeholders with 
interest in and responsibility for aspects of quality assurance will be needed to arrive at a shared 
understanding. We believe the benefits warrant the effort. 


Develop a greater understanding among all relevant actors of the state's interest and role 
in educational quality. There is still some lack of knowledge and appreciation of the state’s 
specific role in quality assurance and improvement. This lack of understanding exists among 
institutional leaders, faculty accreditors, the federal government, and state and system leaders. 
Additional thinking and work are needed to properly articulate and defend this state role and to 
explain why and how states ought to be involved in this work. 


Identify best practices in quality assurance. Additional efforts are needed to identify what works 
in quality assurance and improvement and to diffuse those efforts across states and systems. 
This will require engagement from state and system leaders, college and university leaders, 
faculty, academic researchers, think tanks, policy organizations, and funders. 


Make program review and state authorization meaningful quality assurance processes. 
Essential roles of state agencies and/or system offices—program review and state authorization— 
are often treated as bureaucratic processes and even rubber stamps. Making these processes 
substantive and focused on quality assurance is challenging but necessary. Ideas for improvement 
include: requiring institutions to submit assessable learning outcomes, descriptions of how 
they will meet those outcomes in meaningful ways, plans for faculty development, engaging in 
follow-up reviews of authorized institutions and approved programs, and requiring evidence of 
the institutions’ or programs’ success in accomplishing the approved student learning outcomes. 
Continuing to assess the institution's capacity and resources and other existing requirements will 
likewise be valuable. Further, state and system leaders, researchers, faculty, and other interested 
parties ought to consider what innovations in state authorization and program approval might 
help improve quality assessment and improvement efforts. 


Treat equity as a quality consideration. Among participants and respondents, there appeared 
to be a coalescing around the need to close equity gaps and to treat equity considerations as 
quality considerations. However, better data, research, and political and institutional will are 
needed to properly address quality concerns from an equity perspective. Understanding gaps in 
student resources and opportunities and measuring the ability of institutions to improve higher 
education access and outcomes for underserved students will be critical in considering how 
to address equity gaps. Such examinations need to be done within and across institutions. A 
quality system of higher education needs to be quality for all students, not just well resourced, 
white, and high achieving students. In that regard, states have a particular responsibility for 
looking out for underserved students and ensuring they receive a quality education. Quality 
public institutions should, in the state's eyes, deliberately promote economic mobility and 
opportunity and work to close achievement gaps. Again, collaborative efforts between state 
and system leaders, college and university leaders, academic researchers, faculty, think tanks, 
policy organizations, and funders will be needed. 
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Actively engage faculty and institutional leaders. Ultimately, a quality education is dependent 
on what happens within our colleges and universities and in particular what happens in our 
classrooms. Actively seeking the participation of institutional leaders and faculty will be critical 
to the success of any quality assurance and improvement effort. State agencies and systems 
ought to develop mechanisms for the inclusion of faculty and institutional leaders in the 
state- and system-level quality assurance and improvement efforts, such as working groups 
and advisory boards. They ought to also develop relationships with individual faculty members 
and institutional leaders that allow for state and system leaders to seek ad hoc information, 
reactions, and advice. State agencies and system offices also ought to explore ways they might 
support faculty in their critical role. 


Invest in data, tools, and people. State lawmakers will need to provide state higher education 
agencies and system offices with the financial resources to collect the appropriate data, 
implement the appropriate assessments, conduct the appropriate analysis, properly interpret 
the information, and design and implement the appropriate policies and practices. This will 
require hiring qualified staff and investing in the necessary infrastructure and tools. 


Open lines of communication and real partnership between members of the triad. As described 
and authorized in the federal Higher Education Act, the U.S. Department of Education, approved 
accreditors, and state governments make up the program integrity triad. These three entities 
are supposed to work together to ensure the quality of providers of postsecondary credentials. 
However, the triad has not always functioned appropriately or lived up to its obligations. 
Opening lines of communication, developing agreed upon protocols for information and data 
collection and sharing, developing shared understandings and agreements regarding roles and 
responsibilities, and engaging in more collaborative work and peer learning would all help the 
triad function more effectively. 


CONCLUSION 


We hope the findings from this project will inform the field and generate momentum to improve 
quality assurance efforts at the state and system levels. Much of the real work remains. More 
research, innovation, and thinking are required. Efforts from all responsible and interested parties 
will be required to address the very real challenges facing higher education today and to ensure 
that students receive the quality education they deserve. 
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APPENDIX 


SURVEY AND QUALITATIVE METHODS AND MEETING DESCRIPTIONS 


SURVEY METHODS 


The survey was of state higher education agencies and system offices. Survey results included 
single responses from individual agencies and offices. SHEEO selected the Indiana University 
Center for Postsecondary Research (IUCPR) as the technical contractor to design and implement 
the survey. The survey project was approved by Indiana University’s Institutional Review Board 
(IRB). Following a survey-design meeting held at the SHEEO offices, IUCPR developed a draft 
survey that was circulated among SHEEO, NASH, and Lumina staff involved in the project. Based 
on feedback from this group, IUCPR programmed a pilot survey in Qualtrics that was then 
administered online to a small group of volunteers identified by SHEEO. Feedback from pilot 
participants informed the development of the final survey. Eighty-three agencies and offices 
were invited to participate in the survey, of whom 38 (45.8 percent) agreed to participate. (Two 
declined to participate and received no further reminders.) Of those 38, 22 (26.5 percent of the 
invited group) reached the end of the survey and answered most or all questions. The survey 
asked state agencies and system offices about their organizations’ definitions, activities, and 
experiences concerning quality assurance and improvement in their states, including current 
practices in data gathering and use to catalyze improvements and promote quality outcomes. 


QUALITATIVE METHODS 
SHEEO and NASH conducted nine interviews informed by conversations at the convening in 
Phase 1 and survey results from Phase 2. They conducted these semi-structured telephone 
interviews with 13 SHEEOs, system heads, chief academic officers (CAOs), or agency staff from 
eight states. Interview participants represented five state higher education agencies and four 
university systems. Positions included: 

e Three SHEEOs; 

e Three state higher education agency chief academic officers; 

e Two state higher education agency staff members; 

e One system head; 

e Two system chief academic officers; 

e One system executive vice chancellor; and 

e One system vice provost for planning and effectiveness. 


Interviews were coded and analyzed using an inductive analysis process reflecting rigorous 
qualitative research methods. The code list was derived from major themes identified in Phases 
1 and 2 and was tested and refined on a subset of interviews and reviewed by staff at SHEEO, 
NASH, and Lumina. Findings were triangulated with survey findings. 
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MEETING DESCRIPTIONS 


At the first meeting, the partners convened with a group that included but extended beyond the 
traditional program integrity triad made up of states, the federal government, and accreditors. 
This convening included state higher education executive officers, university system leaders, 
regional and national accreditors, a representative from the U.S. Department of Education, 
and other interested parties to discuss quality assurance and improvement and ideas for 
strengthening the triad. The goals of the meeting included: 


¢ Improving communication among members of the triad; 


e Exploring ways the triad might be strengthened and better assure and improve 
quality in higher education; 


e Exchanging information on the ramifications of recent institutional closures, 
and exploring good practices to help disenfranchised students continue and 
complete their educational programs elsewhere; 


e Exchanging information on the characteristics of other institutions at risk of 
closure and whether the triad, working more closely together, could better 
prepare for or prevent institutional closures; and 


e Understanding what information and data are collected by each member of 
the triad and exploring how this information might be shared for the benefit of 
institutions and students. 


At the second meeting, the partners reconvened with representatives from state higher 
education agencies and system Offices, institutional leaders, academic researchers, policy 
organization staff, accreditors, think tanks, and others to review the findings from the first 
three phases of the project, and share promising quality assurance practices. This group 
also considered changes to current practices and divisions of responsibility regarding quality 
assurance. There were 48 participants. The objectives of the convening were to: 


e Share promising quality assurance practices from various state agencies 
and system offices; 


e Articulate areas of needed reforms in state-level approaches 
to quality assurance; 


e Explore the potential for using existing and emerging sources of data in quality 
assurance efforts and implementing new policies at the state level; and 


e Consider shifts in divisions of responsibilities and new partnerships across 
different actors responsible for improving and assuring quality. 
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